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Abstract 

Background: The main aim was to investigate the effect of using two brief feedback scales in mental health 
out-patient treatment six weeks after starting treatment, compared to treatment as usual. Hypotheses were that use 
of feedback scales would improve treatment alliance and patient satisfaction. 

Methods: An open parallel-group randomised controlled trial was conducted in an out-patient unit in a mental 
health hospital in Central Norway. Eight therapists trained in using the feedback scales in the Partners for Change 
Outcome Management System (PCOMS) treated the intervention group. Seventeen therapists treated the controls, 
providing treatment without using feedback scales. The main outcome measures were treatment alliance and 
patient satisfaction. 

Results: Seventy-five patients participated. There were no differences between the groups in the intention to treat (ITT) 
analyses on alliance (mean difference = 0.08, 95% CI -0.44, 0.59, p = 0.760) or satisfaction (mean difference = 0.24, 95% 
CI -1 .85, 2.32, p = 0.81 9), and no statistically significant differences between the groups in the per protocol (PP, n = 58) 
analyses on alliance (mean difference = 0.32, 95% CI -0.84, 3.16, p = 0.1 37) or satisfaction (mean difference =1.16, 95% 
CI -0.84, 3.1 6, p = 0.248) six weeks after the treatment started. The effect size in favour of the PCOMS group increased 
from 0.07 for alliance and 0.06 for satisfaction in the intention to treat analysis to 0.40 on alliance and 0.31 for 
satisfaction in the per protocol analysis. Among the other outcomes, the PCOMS group had better motivation for 
treatment (estimated mean difference ITT: 0.29, 95% CI 0.00 to 0.57, p = 0.05, PP: 0.28, 95% CI 0.04, 0.52, p = 0.024). 

Conclusion: Six weeks after starting treatment, there were no effects on alliance and satisfaction from using two brief 
feedback scales. Since the per protocol analyses showed higher effect sizes, future investigations in a larger study with 
longer follow-up are warranted. 
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Background 

User participation is highly valued and encouraged in 
the Western world, and there are several proposed bene- 
fits from user participation in health care [1-5]. User 
participation is commonly described on two levels: on a 
system level when users are representatives in boards 
and groups, and on an individual level when patients are 
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being involved in discussions and decisions regarding 
their own treatment. Many have described and defined 
user participation [6-13]. These definitions include 
patients' involvement in their own treatment [7,14], 
patients' right to be involved in decision-making [9], 
patient's implementation and management of their own 
care [12], and a change of patient role from passive reci- 
pients to active participants [15]. These definitions share 
one important aspect: the patient's perspective and influ- 
ence on his or her treatment. This is in line with one of 
the main arguments for user participation in general, 
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that involving the patients' view can improve the quality 
of the health services 

Patients and professionals perceive the quality of health 
care differently, since patients focus more on access, re- 
sponsiveness, good communication, and information, as 
well as appropriate treatment, relief of symptoms, and 
improved health [11]. To measure patient satisfaction 
through surveys is the most common way to assess 
patients' view on health care services [11,16]. Patient satis- 
faction is also one of the most common outcome mea- 
sures when interventions intended to enhance individual 
patient participation are investigated [17,18]. Although 
many have argued that patient satisfaction is insufficiently 
defined [11,16,19], satisfaction is widely assessed in several 
aspects of health care; in the health system as a whole, in 
specific areas of health care i.e. general practice, in specific 
health care organisations such as hospital units, for spe- 
cific clinicians, and for specific treatment approaches [11]. 

Regardless of the health care setting, the relationship 
between the patient and the provider is one of the most 
important factors affecting patient satisfaction [16]. Im- 
proving interpersonal issues is therefore highly recom- 
mended to enhance patient satisfaction [16]. Caring and 
respectful relationships between patient and provider are 
vital for patient satisfaction [20]. The relationship be- 
tween patient and provider has been strongly empha- 
sised in mental health care, and has been described as 
encompassing three parts: a working alliance, a transfer- 
ence configuration, and a real relationship [21,22]. The 
working alliance is considered to be the most fundamen- 
tal for effective treatment [21]. Although the term treat- 
ment or working alliance originated in psychoanalysis, it 
can be generalized to all forms of psychotherapy [23]. 
Treatment or working alliance is described as the bond 
of collaboration and affection between patient and pro- 
fessional [24], and as a concept includes three features: 
that the patient and provider agree on goals, that they 
assign tasks, and that they develop bonds [23]. Research 
has repeatedly shown that there is a consistent relation- 
ship between a strong treatment alliance and a good 
treatment outcome [24]. 

Many have argued for systematically assessing feedback 
from patients during treatment in mental health care [25- 
29]. Such feedback consists of how the patients perceive 
the treatment outcome and how the patients experience 
the treatment session. Systematically assessing feedback 
from patients is part of an increased focus on the patient's 
individual progress during treatment [25]. The main ques- 
tion is thus not whether a treatment works in general, but 
whether this specific treatment works for this specific pa- 
tient [25]. The main argument for assessing feedback is 
twofold. Firstly, when given knowledge of the patient's 
situation at the beginning of treatment, it is possible to 
predict the expected course of change for a successful 



treatment process [25]. Monitoring outcomes during 
treatment makes it possible to detect any deviations from 
the expected course, and this is vital to ensure good treat- 
ment outcomes and to prevent drop-out [30]. Secondly, 
measurements of the patient's perception of the outcome 
and the consultations in every treatment session can be 
used actively to determine whether the current treatment 
is appropriate, whether further treatment is needed, and 
to alert the therapist when the patient is not progressing 
as expected [25,29]. 

Although these arguments are directed at improving 
the outcome from treatment, the process of asking for 
and discussing patient feedback during treatment 
strongly involve the patient's perspective and thereby in- 
crease the patient's participation both in the consulta- 
tions and in the treatment process as a whole. Others 
have previously described interventions where patients' 
rating of health status and quality of life are fed back to 
the providers as enhancing patient participation [17]. 
Given that user participation means including the 
patient's perspective [7,13,14], it would be reasonable for 
the patient to partake in evaluation and the management 
of the treatment process. Since participation in the 
decision-making process is emphasised as a vital part of 
user participation [7,10,13,31], participation would mean 
that the patient is involved in deciding whether the 
current treatment should continue or be altered. Ex- 
changing information through a dialogue between pa- 
tient and therapist has also been emphasised as a core 
aspect of user participation [13]. 

Increasing participation has also been described as a 
redistribution of power [32]. Inviting the patient to par- 
take in evaluation and decision-making is a concrete 
way to involve the patient more, and would transfer 
some power over the treatment process from the profes- 
sional to the patient. To assess and collect comprehen- 
sible feedback data, and to monitor these data during 
the treatment process, makes it feasible for the patient 
and provider to evaluate and discuss treatment progress 
as equal partners. Concretely assessing feedback system- 
atically invites the patient to exchange information with 
the provider, to evaluate the treatment effect and the 
consultation, to discuss potential improvements, and to 
determine whether the treatment approach is appropri- 
ate or should be altered. Systematically assessing and 
monitoring feedback from patients on the treatment 
outcome is thus a practical tool of user participation on 
the individual level. 

Effort has been made to simplify the process of collect- 
ing and using patient feedback data, and to make it prac- 
tical for both patients and professionals. Feedback 
systems have been developed to monitor the treatment 
progress from the patients' point of view, i.e. the Out- 
come Questionnaire-45 (OQ-45) [33] and the Partners 
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for Change Outcome Management System (PCOMS) 
[30]. These systems use standardised feedback scales to 
collect the patient's views on outcomes in every consult- 
ation, and the results are discussed by the patient and 
the therapist during the treatment process [29]. A meta- 
analysis of studies on the effect of the OQ-45 and the 
PCOMS showed that patients who experienced little 
progress during treatment were detected through sys- 
tematically assessing feedback, and they obtained better 
treatment outcomes than patients did where feedback 
was not assessed [29]. 

While the OQ-45 includes a rather extensive scale 
with 45 items, the Partners for Change Outcome Man- 
agement System (PCOMS) includes two short feedback 
scales with four items each: the Outcome Rating Scale 
(ORS) and the Session Rating Scale (SRS) [30]. The ORS 
is used at the beginning of each treatment session to as- 
sess the patients' rating of treatment outcome last week, 
or since the last session. The SRS is used at the end of 
each session to assess the patients' rating of the current 
session. Only a few randomised controlled trials have 
investigated the effect of using the brief feedback scales 
in the PCOMS [29]. These trials have investigated effects 
in mental health treatment [34], in couples therapy 
[35,36], and in training and supervision of psychology 
students [37]. All four trials focused on treatment out- 
come measured by the ORS and found that using the 
PCOMS scales during treatment was superior to treat- 
ment as usual. 

Aim and hypotheses 

To discuss the patient's perception of the treatment out- 
come in some manner is widely acknowledged as an im- 
portant and inherent part of all treatment for mental 
health problems. Using concrete feedback scales where 
the patient is repeatedly asked to evaluate and rate the 
treatment and the consultations, is a more definite way 
to assess and collect the patient's views. Scales are a 
quantified form of patient feedback which can be used 
in further discussions on how to improve the treatment. 
Collecting feedback scales could also be considered a 
strong and definite signal that the patient's views matter. 
It would therefore be of interest whether routinely using 
feedback scales to assess and monitor the patient's views 
increases the alliance and satisfaction in early phases of 
treatment. Treatment including the use of feedback 
scales should therefore be compared to usual treatment, 
where feedback scales are not used. No studies so far 
have investigated the short-term effect of using brief 
feedback scales, such as the PCOMS, on treatment alli- 
ance and patient satisfaction. 

The aim was therefore to investigate the short-term 
effects on treatment alliance and patient satisfaction 
from using the PCOMS scales (ORS and SRS) in out- 



patient mental health treatment, compared to treatment 
without using feedback scales (treatment as usual). 

One of the arguments for using feedback scales is that 
a strong treatment alliance increases the possibility for a 
good treatment outcome [26]. Treatment alliance is con- 
sidered to be an important predictor for outcome in 
mental health care [30,38,39], and it is found that the 
degree of treatment alliance can be established after only 
a few consultations [38]. Hypothesis 1 was therefore that 
the use of the PCOMS scales (ORS and SRS) would lead 
to stronger treatment alliance than treatment as usual, 
six weeks after starting treatment. In this study we 
defined treatment alliance as the patient's perception of 
a relational bond and good collaboration with the ther- 
apist [40]. 

It has also been argued that using the ORS and SRS 
scales to ask patients for feedback during treatment helps 
foster a cooperative and accountable relationship between 
patient and professional [26]. The relationship between 
patient and provider is the most influential factor on pa- 
tient satisfaction [41,42], and it would be reasonable to be- 
lieve that the use of feedback scales would lead to 
improved patient satisfaction. Hypothesis 2 was therefore 
that the use of the PCOMS scales (ORS and SRS) would 
lead to a higher degree of patient satisfaction than treat- 
ment as usual, six weeks after starting treatment. 

Secondary outcomes were mental health symptoms, 
patient activation, health-related quality of life, patient 
motivation, and patient participation. 

Methods 

This was an open, randomised parallel-group controlled 
trial performed according to the principles of the Hel- 
sinki Declaration. The Regional Committee for Medical 
and Health Research Ethics, Central Norway, and the 
hospital's management approved of the study. Partici- 
pants had to sign an informed consent and were 
informed that they could withdraw during the study. 
Data collection was conducted from February 2010 to 
March 2011. 

The study was carried out in an out-patient unit in a 
mental health hospital in Central Norway (Trondheim). 
The hospital, a part of St. Olav's Hospital Trust, covers a 
catchment area of 96,000 people, with urban and semi- 
rural areas including parts of a large Norwegian city. The 
out-patient unit treats patients with all types of mental 
health diagnoses that do not require hospitalisation. 

Participants 

All patients offered treatment at the out-patient unit be- 
tween six weeks and three months after referral were eli- 
gible. The lower limit of six weeks was needed to allow 
for baseline assessment, randomisation, and treatment 
allocation in the units. There were no exclusion criteria. 
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All therapists at the out-patient unit who provided and 
were responsible for individual treatment were eligible 
for participating in the study. 

Recruitment 

Invitation to participate was mailed to the patients to- 
gether with the treatment approval letter. One week 
after the invitation was mailed an employee at the de- 
partment or the 1 st author (MBR) phoned the patients 
asking if they wanted to participate in the study. Those 
who accepted were invited to meet with the 1 st author. 
During this meeting, oral and written information was 
given and a written consent was signed. Three patients 
wanted to participate without meeting with the re- 
searcher. They were informed by phone, received the 
written consent by mail, and returned it by mail. 

All 29 therapists in the out-patient unit who provided 
and were responsible for individual treatment were 
invited to participate in the study. The invitation 
informed that they must be willing to receive training in 
use of the PCOMS scale on two specific days. Ten thera- 
pists volunteered to participate in the study, but only 8 
could participate in the training at the given times. The 
two therapists who weren't able to participate in the 
training sessions and six other therapists were recruited 
to participate in treating the treatment as usual group. 
The six therapists who had not volunteered were asked 
by the unit manager to participate. Due to administrative 
turn-over and other unanticipated difficulties in allocat- 
ing control patients to therapists during the trial, the 
group of treatment as usual therapists had to be 
extended to 17 during the study. In total 86% of all 
therapists in the out-patient units at the hospital who 
provided individual treatment participated in the study. 

Intervention 

The intervention therapists were trained to administer 
the feedback system Partners for Change Outcome Man- 
agement System (PCOMS) [30] during the treatment 
they usually provide. PCOMS therapists received 12 h of 
training during two days, with four weeks apart, with re- 
spectively eight and four hours of training. In addition, 
the therapists could contact the instructor at their own 
will. The training the PCOMS therapists received was 
similar to previous studies [34-37], and in accordance 
with training given for clinical purposes to improve pa- 
tient functioning and progress [43]. Training was given 
by an experienced instructor who did not participate in 
the research group. The instructor has been trained by 
the founders of the PCOMS, and has extensive experi- 
ence both from using the system in clinical practice and 
from teaching other clinicians. Training included the ra- 
tionale for the use of patient feedback and feedback 
scales, the practical use of the feedback scales, and how 



to incorporate the use of data from the scales in the 
treatment process. 

The use of the PCOMS consisted of administering two 
feedback scales in every treatment session, one at the be- 
ginning of the session (the Outcome Rating Scale, or 
ORS), and one at the end (the Session Rating Scale, or 
SRS) [30]. Completing and scoring each scale takes less 
than one minute. Both scales consist of four questions 
scored on a 10 cm scale. In the ORS the patients rate their 
own functioning during the last week, or since the last 
treatment session, individually, interpersonally, socially, 
and generally. On the SRS, the patients rate the current 
session on relations with the therapist and the degree of 
agreement on goals, methods, and treatment approach. 

Based on the patients' initial ORS score, a progress 
curve was produced with a dotted line representing the 
expected trajectory of change for patients with similar 
ORS score in the first session [30] . The therapist had been 
trained to use this curve together with the patient to 
evaluate treatment progress. They had also learned to dis- 
cuss the SRS scores with the patients to establish what 
was working in the session and what could be improved. 
The intervention thus consisted of systematically using 
the ORS and SRS scales to assess feedback from the pa- 
tient on treatment outcome and the quality of the session. 

To ensure fidelity in the PCOMS group, the patient 
feedback scales were collected after each consultation and 
the numbers of scales were compared to the total number 
of consultations registered on each participant in the 
clinic's administrative data system. The controls received 
treatment as usual. To fortify fidelity in the treatment as 
usual group the therapists were repeatedly instructed to 
avoid using any feedback scales during treatment. 

Both PCOMS and treatment as usual therapists were 
free to choose treatment approaches for their patients, 
and no recommendations or limitations were made, due 
to the study. The therapists thus chose the treatment ap- 
proach they considered appropriate for each patient. 
The therapists working in the out-patient units in gen- 
eral mostly use psychotherapy, cognitive behavioural 
therapy, and some pharmacotherapy. 

Data collection/outcome measures 

At baseline the participants completed a questionnaire 
on background information (Table 1). Outcome data 
were completed at baseline and six weeks after treat- 
ment started. Main outcome measures after six weeks of 
treatment were Treatment Alliance Scale (TAS) [40] and 
Client Satisfaction Questionnaire (CSQ) [44]. 

The Treatment Alliance Scale (TAS) is a 10-item scale 
measuring patients' experience of alliance with therapists 
[40], providing a total score from 0 to 6 (strongest alli- 
ance). TAS was translated to Norwegian for the present 
study and we replaced "treatment team" with "therapist" 
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Table 1 Patients: total sample, intervention, and control 



Variables 


Total sample 


Intervention 


Control 




N = 75 


n = 37 


n = 38 




N (%) 


n (%) 


n (%) 


Female 


47 (62.7%) 


26 (703%) 


21 (55.3%) 


Age (Mean (Median, Range)) 


29.9 (25, 1 8-70) 


30.5 (26, 1 8-64) 


29.2 (25, 20-70) 


Living alone 


23 (30.7%) 


9 (24.3%) 


14 (36.8%) 


Can confide in two or more persons 


54 (72.0%) 


30 (81.1%) 


24 (63.2%) 


Highest level of education 








- Primary and lower secondary school 


13 (17.3%) 


9 (24.3%) 


4 (10.5%) 


- Upper secondary school 


32 (42.7%) 


1 2 (32.4%) 


20 (52.6%) 


- University 


30 (40.0%) 


1 6 (43.2%) 


14 (36.8%) 


Working 


1 7 (22.7%) 


11 (29.7%) 


6 (15.8%) 


In education 


24 (32.0%) 


11 (29.7%) 


13 (34.2%) 


Currently using medication for mental health problems 


32 (42.7%) 


14 (37.8%) 


18 (47.4%) 


Previous treatment for mental health problems 


59 (78.7%) 


29 (78.4%) 


30 (78.9%) 


Previously hospitalized for mental health problems 


14 (18.7%) 


8 (21.6%) 


6 (15.8%) 



Numbers are N/n (%) unless otherwise stated. 



in all items to make it suitable for out-patients. TAS has 
solid psychometric properties [40]. The Treatment Alli- 
ance Scale was chosen since it includes questions consid- 
ered highly relevant; whether the patient experienced the 
therapist as respectful, understanding, attentive, helpful, 
and accessible. The questions also included the patients' 
rating of collaboration, shared understanding, active par- 
ticipation, and whether the treatment would be helpful. 

The Client Satisfaction Questionnaire-8 (CSQ-8) mea- 
sures patient satisfaction [44]. The questionnaire includes 
eight items, providing a total score from 8 to 32 (highest 
satisfaction). The CSQ-8 has good psychometric proper- 
ties [44,45], and was translated to Norwegian for the 
present study. 

Several outcomes were added to investigate various 
aspects: 

Since the use of the PCOMS are supposed to improve 
treatment results in mental health care, the Behaviour 
and Symptom Identification Scale 32 (BASIS-32) [46] 
was included. BASIS-32 is a 32 item questionnaire meas- 
uring mental health symptoms and functioning and pro- 
vides a total score from 0 to 4 (worst mental health 
symptoms) [46]. BASIS-32 was translated to Norwegian 
for the present study. 

Patient activation is a central aspect of an active patient 
role where patients and providers are more equal partners 
[47]. Patient Activation Measure (PAM) was therefore 
used to measure patient activation [47], providing a total 
PAM score ranging from 0 to 100 (highest activation). 
The validated Norwegian version was used [48] . 

Improved treatment results and patient activation 
would also imply improved health-related quality of life 



in patients, and Short Form-12v2 (SF-12v2) was there- 
fore used to measure the patient's self-reported health- 
related quality of life [49]. SF-12v2 is a recognised and 
much used health-related quality of life measurement. 
The 12 item scores are calculated into two total scores: 
The Mental Component Score (MCS), reflecting mental 
health-related quality of life, and the Physical Compo- 
nent Score (PCS), reflecting physical health-related qual- 
ity of life. The total scores range from 0 to 100 (best 
quality of life). We used the Norwegian version of 
SF12v2 (SF-12v2™ Health Survey, 2004, Health Assess- 
ment Lab, Medical Outcomes Trust and Quality Metric 
Inc.). 

Since all previous studies [34-36] investigating the ef- 
fect of PCOMS have used the Outcome Rating Scale 
(ORS) as primary outcome, we included the ORS and 
the Session Rating Scale (SRS) as secondary outcome 
measures [30]. The total scores for each scale range from 
0 to 40 (best). The Norwegian versions were used. 

To measure user participation, 14 statements about 
participation and motivation, developed by the authors, 
were added. The statements were based on the literature 
and our own work on how patients and health personnel 
define patient participation [13]. Baseline data were used 
to conduct an exploratory factor analysis. Using varimax 
rotation, two factors were found to have an eigenvalue 
above 1.0, explaining nearly 63% of the variance. Three 
statements on motivation for treatment constituted one 
factor - "Patient motivation" (PM). Examples of state- 
ments on motivation are "I want to make a large effort 
to improve," and "It is very important for me that my 
mental health problems improve". The remaining 11 
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statements constituted the second factor - "Patient par- 
ticipation" (PP). Examples of statements on participation 
are "I am treated with respect from my therapist and 
others I have contact with during my treatment," and 
"My therapist is interested in hearing my opinion." Each 
statement was scored from 0 (strongly disagree) to 6 
(strongly agree), and the total score was the mean of all 
statements (0 to 6, where 6 is the strongest motivation 
and most participation). 

For the translations of TAS, CSQ, and BASIS-32, two 
persons independently translated the questions from 
English to Norwegian, and two other persons independ- 
ently translated them from Norwegian to English. Lack 
of accordance was discussed until consensus was 
reached. The Norwegian versions were tested on volun- 
teers. Effort was made to keep the Norwegian versions 
similar to the originals. 

Power calculation and sample size 

Due to the lack of similar studies on treatment alliance, 
the sample size was determined using the assumed clin- 
ically significant difference in the BASIS-32 question- 
naire (mental health symptoms and functioning). To 
detect a clinically significant difference between the 
groups' BASIS-32 scores of 0.5 with a standard deviation 
of 0.8 [50], 32 participants were needed in each group 
(power 0.8, alpha 0.05). No interim analyses were 
planned or conducted. 

Randomisation and allocation 

After completing baseline data, participants were rando- 
mised to either PCOMS or treatment as usual, using the 
university's internet based computerised randomisation ser- 
vice. There was no stratification or block randomisation. 
Patients were subsequently allocated to a PCOMS therapist 
or treatment as usual therapist by the unit manager. Thera- 
pists could thus not influence which patients they treated. 

Blinding 

This was an open study and no blinding was performed. 
Statistical methods 

For all questionnaires, only those with at least 50% of the 
items answered valid were scored. Missing values were 
replaced with last value (baseline) carried forward, or if no 
baseline values were available, missing values were 
replaced with the mean value of the group the patient was 
randomised to. Information on the number of consulta- 
tions for three patients in the intervention group and nine 
in the control group was not accessible from the clinic's 
administrative data system. These missing data were not 
replaced. Analyses both with and without missing data 
were conducted and the results were the same. 



Both intention to treat and per protocol analyses were 
conducted [51]. The pre-defined criterion for per proto- 
col analysis was that patients should have attended at 
least three consultations during six weeks from the ini- 
tial consultation. This would ensure that all patients 
received a minimum of treatment. In addition, the 
PCOMS group should have used the feedback scales in 
at least 2/3 of all consultations to ensure that the 
PCOMS group received the intervention. 

Between group differences for TAS, CSQ, SRS, and PP 
were analysed using two-tailed independent t-tests. Be- 
tween group differences for the other outcomes were 
analysed using analysis of covariance (ANCOVA) with 
the baseline value as a covariate. Within group differ- 
ences (from baseline to six weeks) were calculated using 
two-tailed paired t-tests. A significance level of 5% (p < 
0.05) was chosen to calculate confidence intervals (95% 
CI). Analyses were done with SPSS 17.0 for Windows 
(SPSS Inc., Chicago, IL). 

Results 

Participant flow 

The flow of participants during recruitment and study is 
described in Figure 1. Three hundred and ninety- five 
out-patients were eligible and received an invitation let- 
ter. Eighty-seven volunteered to participate and 12 with- 
drew before randomisation. A total of 75 patients were 
randomised; 37 were put in the PCOMS group and 38 
in the treatment as usual group. 

Baseline data 
Patients 

In the total sample 63% were female and the mean age 
was 30 years (Table 1). 31% lived alone, and 72% had 
two or more persons they could confide in. 79% had pre- 
vious treatment experience for mental health problems, 
and 43% were currently using medication for such 
problems. 

Age, gender, and referral diagnoses were collected con- 
secutively for 40% (n = 157) of the total group of patients 
who were eligible and invited to, but did not participate. 
Due to data registration restrictions, we were not able to 
collect data for all non-participating patients. The mean 
age was 28.4 (range 18-71), and 58% were female. Diag- 
noses at referral were also similar with the patients who 
participated, with anxiety and depression as the main 
diagnosis (data not shown). 

Therapists 

Eight therapists treated the patients in the PCOMS 
group, while 17 treated the patients in the treatment as 
usual group (Table 2). Four of the 17 therapists in the 
treatment as usual group did not respond to a question- 
naire about therapists' characteristics, and these missing 
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Assessed for eli 


gibility (n=395) 




Excluded (n=320) 
-Administrational reasons (n=22) 
-Missing phone number (n-78) 
-Unable to reach (n=108) 
-Declined to participate (n=100) 
-Withdrew before 
randomization (n=12) 








Randomized (n=75) 









a - 
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Figure 1 Flow chart study. 



Allocated to intervention (n=37) 
Received intervention (n=26) 
Did not receive intervention, 

- allocated to control group (n=4) 

- never received treatment (n=l) 

- scales were not used in at least 2/3 of 
the sessions (n=2) 

-changed therapist to control after 2 
sessions (n=l) 

-received less than 3 sessions (n=3) 



Therapists (n=8) 

Number of patients treated by each 
therapist (median=5) 
[IQR=1.5-7,Range=l-7] 



Lost to follow-up: 

- lost contact after baseline (n=l) 



Analyzed 

- intention to treat (n=37) 

- per protocol (n=26) 



Allocated to control (n=38) 
Received control treatment (n=32) 
Did not receive control, 

- never received treatment (n=2) 

- received less than 3 sessions (n=4) 



Therapists (n=17) 
Number of patients treated by each 
therapist (median=l .5) 
[IQR=l-2.75, Range=l, 6] 



Lost to follow-up: 

- lost contact after baseline (n=l) 

- failed to return questionnaire (n=2) 



Analyzed 

- intention to treat (n=38) 

- per protocol (n=32) 



values were not replaced. 64% of the therapists were fe- 
male, 76% were psychologists and 24% were psychiatric 
nurses. The mean age was 41 years, they had 1-16 years 
of experience with out-patient mental health treatment, 



and 1-25 years of experience from mental health care in 
general. 43% had heard about the PCOMS before, and 
9.5% had previously attended a course in using the 
PCOMS in clinical work. The number of therapists 



Table 2 Therapists; total sample, intervention, and control 



Variables 


Total sample 


Intervention 


Control 


P- 




N = 25 


n = 8 


n = 17 


value 




N (%) 


n (%) 


n (%) 




Female 


16 (64%) 


5 (62.5%) 


1 1 (64.7%) 


0.915 


Age (Mean (Median, Range)) 


414 (38, 30-62) 


39 (34.5, 31-62) 


42.9 (41, 30-61) 


0.402 


Education 








0.920 


- Psychiatric nurse 


5 (23.8%) 


2 (25%) 


3 (17.6%) 




- Psychologist 


16 (76.2%) 


6 (75%) 


1 0 (58.8%) 




No of years worked in mental health care (Mean (Median, Range)) 


9.9 (9, 1-25) 


7.4 (5.5, 1-16) 


11.4 (10, 1-25) 


0.220 


Wo of years worked in out-patient unit (Mean (Median, Range)) 


4.6 (3, 1-16) 


3.4 (3, 1 -8) 


5.4 (4, 1-16) 


0.292 


Heard about PCOMS prior to study (yes) 


9 (42.9%) 


3 (37.5%) 


6 (35.3%) 


0.697 


Attended course in PCOMS prior to study (yes) 


2 (9.5%) 


1 (12.5%) 


1 (7.7%) 


0.716 



Numbers are N/n (%) unless otherwise stated. 

For categorical variables the groups have been compared using Pearson's chi square tests. 
For continuous variables the groups have been compared using two-tailed independent t-tests. 
Four treatment as usual therapists did not return the questionnaire on these variables. 
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participating was 86% of the total number of therapists 
in the outpatient unit at the hospital who provided indi- 
vidual treatment. The two groups of therapist were simi- 
lar regarding gender, education, age, and work 
experience. The PCOMS therapists treated a mean of 
4.4 patients each (median 5, range 1-7), while the treat- 
ment as usual therapists treated a mean of 2.4 patients 
each (median 2, range 1-6). 

Implementation of intervention 

During the six weeks, the patients in the PCOMS group 
had an average of 3.8 (range 1-9) consultations. The 
patients in the treatment as usual group had an average 
of 3.8 (range 1-10) consultations. In the PCOMS group 
the scales were used in 92% of all consultations. 

Comparison between the groups 

The differences between the PCOMS group and treat- 
ment as usual group on all outcome measures are pre- 
sented in Table 3. 

There were no differences between the groups on 
treatment alliance (TAS) (mean difference = 0.08, 95% CI 
-0.44 to 0.59, p = 0.760) and patient satisfaction (CSQ) 
(mean difference = 0.24, 95% CI -1.85 to 2.32, p = 0.819) 
six weeks after the treatment started. 

There was a higher score on motivation for treatment 
(PM) in the PCOMS group (estimated mean difference = 
0.29, 95% CI 0.00 to 0.57, p = 0.05). There were no differ- 
ences on the other outcomes. 

Changes within the groups 

The changes in means from baseline to six weeks after 
starting treatment, for both the PCOMS group and 
treatment as usual group, are presented in Table 3 (last 
2 columns). Since it was not possible to complete the 
outcomes on alliance (TAS), satisfaction (CSQ), treat- 
ment sessions (SRS), and patient participation (PP) be- 
fore starting treatment, no changes within groups could 
be calculated for these variables. Two outcomes had im- 
provement in the PCOMS group; patient activation 
(PAM, mean diff=3.22, 95% CI 0.70 to 5.74, p = 0.014) 
and patient functioning (ORS, mean diff = 3.53, 95% CI 
0.99 to 6.06, p = 0.008). There were no differences in the 
treatment as usual group. 

Per protocol analyses 

Per protocol analyses were conducted only on partici- 
pants who received the intervention they were rando- 
mised to according to protocol. The predefined per 
protocol criteria gave 26 participants in the intervention 
group and 32 in the control group. Reasons for not 
meeting the criteria in the PCOMS group were being 
mistakenly allocated to the control group (four patients), 
never starting treatment (one patient), not using scales 



in at least 2/3 of the treatment sessions (two patients), 
changing therapist to control after two sessions (one pa- 
tient), and receiving less than three sessions (three 
patients). Reasons for not meeting the criteria in the 
treatment as usual group were never starting treatment 
(two patients) and receiving less than three treatment 
sessions (four patients). Patients meeting per protocol 
criteria received an average of 4.0 treatment sessions 
during six weeks. 

The per protocol analyses showed no differences be- 
tween the PCOMS group and the treatment as usual 
group on alliance or satisfaction, six weeks after starting 
treatment (Table 4), but, compared to the intention to 
treat analysis, the absolute values tended to change in 
favour of the PCOMS group. The effect size [52] 
increased from 0.07 (intention to treat, Table 3) to 0.40 
(per protocol, Table 4) for treatment alliance and from 
0.06 to 0.31 for patient satisfaction. 

The PCOMS group had higher scores than the treat- 
ment as usual group on motivation for treatment (PM) 
(estimated mean diff =0.28, 95% CI 0.04 to 0.52, p = 
0.024), and near significant higher scores on evaluation 
of consultations (SRS) in the PCOMS group (mean diff = 
3.15, 95% CI 0.07 to 6.37, p = 0.055). 

For within group analysis, PAM and ORS improved 
significantly in the PCOMS group, as in the intention to 
treat analysis. 

Discussion 

Results 

Six weeks after starting treatment, no significant effects 
on treatment alliance and patient satisfaction from using 
the PCOMS scales were found. Both hypotheses were 
thus rejected. The PCOMS group had higher motivation 
for treatment compared with the treatment as usual 
group, and there were improvements within the PCOMS 
group of patient activation (PAM) and patient function- 
ing (ORS). 

Strengths and limitations 

The main strength of this study is that it is the first ran- 
domised controlled trial on the effect on treatment alli- 
ance and patient satisfaction of systematically using the 
two PCOMS scales in treatment sessions. The study was 
conducted in a natural setting. The sample of patients 
participating in this study was similar in gender, age, and 
referral diagnoses to a larger sample from this hospital 
out-patient unit, indicating that the study sample were 
representative of those referred to out-patient mental 
health treatment. Although the therapists initially self- 
selected to participate, 86% of all eligible therapists at 
the out-patient units eventually participated in the study. 
The results therefore have strong external validity. The 
mean ORS score at baseline for the participants in the 
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Table 3 Intention to treat analyses 



Outcomes 


Group 


Baseline 


Between groups at 6 weeks A) 




Within groups (6 weeks-baseline) B) 






Mean (SD) 


Mean (SD) 


Diff. (95% CI) 


p-value 


Diff (95% CI) 


p-value 


TAS 


INT 
CTRL 




4.63 (1.1) 
4.55 (1.1) 


0.08 (-0.44, 0.59) 


0.760 






CSQ 


INT 
CTRL 




24.42 (5.0) 
24.18 (4.0) 


0.24 (-1.85, 2.32) 


0.819 






BASIS-32 f 


INT 


1 .05 (0.5) 


1 .00 (0.5) 


0.02 (-0.20, 0.24) 


0.882 


-0.05 (-0.21, 0.12) 


0.551 




CTRL 


1.23 (0.5) 


1 .08 (0.6) 






-0.15, (-0.33, 0.03) 


0.107 


PAM 


INT 


35.36 (9.1) 


38.58 (7.7) 


1 .58 (-2.28, 5.44) 


0.417 


3.22 (0.70, 5.74)* 


0.014 




CTRL 


36.09 (9.1) 


37.34 (10.8) 






1 .25 (-2.43, 4.94) 


0.495 


MCS (SF-12) 


INT 


38.19 (8.4) 


40.63 (9.2) 


0.03 (-3.5, 3.6) 


0.989 


2.44 (-0.49, 5.38) 


0.100 




CTRL 


38.08 (5.9) 


40.55 (7.9) 






2.47 (-0.21, 5.16) 


0.070 


PCS (SF-12) 


INT 


48.05 (7.6) 


4910 (8.9) 


0.92 (-2.05, 3.89) 


0.539 


1.04 (-1.06, 3.14) 


0.321 




CTRL 


48.29 (7.9) 


48.35 (8.7) 






0.69 (-2.22, 2.36) 


0.952 


ORS 


INT 


1 8.64 (8.2) 


22.17 (9.2) 


1.73 (-1.89, 5.35) 


0.344 


3.53 (0.99, 6.06)* 


0.008 




CTRL 


1 6.83 (6.7) 


1 9.33 (8.8) 






2.51 (-0.41, 5.43) 


0.090 


SRS 


INT 
CTRL 




31.88 (7.9) 
3043 (8.3) 


1.46 (-2.28, 5.19) 


0.440 






PM 


INT 


5.34 (0.8) 


5.59 (0.6) 


0.29 (0.00, 0.57)* 


0.050 


0.25 (-0.06, 0.57) 


0.113 




CTRL 


5.31 (0.8) 


5.30 (0.8) 






-0.02 (-0.23, 0.19) 


0.874 


PP 


INT 
CTRL 




4.43 (1.0) 
4.52 (0.9) 


-0.09 (-0.52, 0.34) 


0.663 







Comparison of outcomes between the intervention group (INT, n = 37) and control group (CTRL, n = 38) six weeks after treatment start, and within each group 
from baseline to six weeks. 

TAS = Treatment alliance score, CSQ = Client satisfaction questionnaire, BASIS-32 = The behavior and symptom identification scale 32, PAM = Patient activation 
measure, MCS (SF-12) = Mental Component Score (Short form-12), PCS (SF-12) = Physical Component Score (Short form-12), ORS = Outcome rating scale, SRS = 
Session rating scale, PM = Patient motivation, PP = Patient participation. 

A) The between-group mean differences for TAS, CSQ, SRS and PP have been calculated using two-tailed independent t-tests. The between-group estimated 
mean differences for BASIS-32, PAM, SF12, ORS, and PM have been calculated using analysis of covariance (ANCOVA), adjusted for baseline values. 

B) The within group differences have been calculated using two-tailed paired f-test. 

f Decrease in score indicates improvement (less mental health symptoms) for BASIS-32. For the other outcomes, increased score indicates improvement. 
* p-value <0.05. 



study was 18.8. Since this score is well below 25, which 
is considered the cut-off for those in need of treatment 
[30], this was a suitable sample from mental health care. 

The high adherence to the use of the scales, which was 
used in 92% of the consultations, confirms that the inter- 
vention was delivered as planned. To fortify fidelity to the 
study design, the treatment as usual therapists were re- 
peatedly instructed to avoid using any feedback scales in 
the consultations, but the consultations were not video or 
audio taped. We found no indications that feedback scales 
were used in the treatment as usual group. 

The training the PCOMS therapists received was simi- 
lar to previous studies [34-37], and in accordance with 
training given for clinical purposes to improve patient 
functioning and progress [43]. According to Miller and 
colleagues [30], patients using PCOMS should have a 
change of at least 5 points on the ORS score to show re- 
liable change after three consultations. In the present 
study, the mean change after three consultations was 6.6 



points, indicating that the sample followed the expected 
treatment progress. 

The size of the study might be a limitation. Given the 
results for treatment alliance in the per protocol analysis 
(difference in TAS = 0.32, standard deviation 0.8) and 
that additional patients had got the same results, a total 
of 113 patients in each group would be needed to get a 
result with a p-value < 0.05 (power 0.8, alpha 0.05). 

Cronbach's alpha was calculated and found satisfactory 
for the outcome measures that were translated to Norwe- 
gian for this study. The results were 0.89 for BASIS-32 
(mental health symptoms and functioning), 0.93 for TAS 
(treatment alliance), and 0.96 for CSQ (patient satisfaction). 

Alliance and satisfaction 

We hypothesised that, six weeks after starting treatment, 
treatment alliance and patient satisfaction would be 
higher in the PCOMS group than the treatment as usual 
group. Both hypotheses were rejected. That there were 



Rise et al. BMC Health Services Research 2012, 12:348 
http://www.biomedcentral.com/1472-6963/12/348 



Page 10 of 12 



Table 4 Per protocol analyses 


Outcomes 


Group 


Baseline 


Between groups at 6 weeks A) 




Within groups (6 weeks-baseline) B) 






Mean (SD) 


Mean (SD) 


Diff. (95% CI) 




Diff (95% CI) 


p-value 


TAS 


INT 
CTRL 




5.01 (0.8) 
4.69 (0.8) 


0.32 (-0.11,0.75) 


0.137 






CSQ 


INT 
CTRL 




25.59 (4.0) 
24.43 (3.6) 


1.16 (-0.84, 3.16) 


0.248 






BASIS-32 f 


INT 


1.10 (0.5) 


1 .03 (0.5) 


0.04 (-0.22, 0.30) 




-0.07 (-0.27, 0.14) 


0.515 




CTRL 


1 .27 (0.6) 


1 .09 (0.6) 




0.766 


-0.18 (-0.37, 0.01) 


0.069 


PAM 


INT 


35.17 (6.4) 


39.43 (7.0) 


2.53 (-2.04, 7.10) 


0.272 


4.26 (1.71, 6.81)* 


0.002 




CTRL 


36.17 (8.8) 


37.29 (10.5) 






1.12 (-3.15, 5.40) 


0.596 


MCS (SF-12) 


INT 


37.41 (7.5) 


39.94 (7.8) 


-0.29 (-4.21, 3.63) 


0.883 


2.54 (-0.40, 5.48) 


0.088 




CTRL 


37.37 (6.1) 


40.21 (8.5) 






2.84 (-0.23, 5.92) 


0.069 


PCS (SF-12) 


INT 


47.46 (8.4) 


48.84 (9.9) 


1 .42 (-2.20, 5.04) 


0.434 


1.38 (-1.48, 4.24) 


0.330 




CTRL 


48.09 (8.6) 


47.90 (9.0) 






-0.18 (-2.70, 2.34) 


0.883 


ORS 


INT 


1 7.55 (7.0) 


22.07 (9.0) 


2.68 (-1.74, 7.09) 


0.229 


4.52 (1.31, 7.73)* 


0.008 




CTRL 


16.16 (7.0) 


1 8.63 (9.2) 






2.47 (-0.94, 5.89) 


0.150 


SRS 


INT 
CTRL 




34.68 (5.6) 
31.53 (6.4) 


3.15 (0.07, 6.37) 


0.055 






PM 


INT 


5.38 (0.8) 


5.65 (0.5) 


0.28 (0.04, 0.42) 


0.024 


0.27 (-0.04, 0.57) 


0.081 




CTRL 


5.32 (0.9) 


5.34 (0.7) 






0.02 (-0.14, 0.19) 


0.788 


PP 


INT 
CTRL 




4.73 (0.7) 
4.66 (0.6) 


0.07 (-0.29, 0.42) 


0.706 







Comparison of outcomes between the intervention group (INT, n = 26) and control group (CTRL, n = 32) six weeks after treatment start, and within each group 
from baseline to six weeks. 

TAS = Treatment alliance score, CSQ = Client satisfaction questionnaire, BASIS-32 = The behavior and symptom identification scale 32, PAM = Patient activation 
measure, MCS (SF-12) = Mental Component Score (Short form-12), PCS (SF-12) = Physical Component Score (Short form-12), ORS = Outcome rating scale, SRS = 
Session rating scale, PM = Patient motivation, PP = Patient participation. 

A) The between-group mean differences for TAS, CSQ, SRS and PP have been calculated using two-tailed independent t-tests. The between-group estimated 
mean differences for BASIS-32, PAM, SF12, ORS, and PM have been calculated using analysis of covariance (ANCOVA), adjusted for baseline values. 

B) The within group differences have been calculated using two-tailed paired f-test. 

f Decrease in score indicates improvement (less mental health symptoms) for BASIS-32. For the other outcomes, increased score indicates improvement. 
* p-value <0.05. 



no effects on alliance or satisfaction in the present study 
questions if the alleged increase in alliance when using 
the PCOMS scales is due to treatment in general or 
feedback techniques embedded in treatment, and not 
the use of the scales. The observation that the differ- 
ences between the groups in the per protocol analysis 
(where only those who used the scales as intended were 
included) was more in favour of the PCOMS group than 
in the intention to treat analysis, warrants a larger study 
to draw a more firm conclusion. 

The instrument used to measure treatment alliance in 
this study (Treatment Alliance Scale (TAS)) has been rela- 
tively little used and tested so far, but was chosen since it 
is a validated questionnaire which is easy to complete. 
More used instruments like The Revised Helping Alliance 
Questionnaire [53] and The Working Alliance Inventory 
[54] could have been chosen. However, a meta-analysis on 
alliance questionnaires reported no difference in ability to 
measure alliance across instruments [24]. 



Since the Session Rating Scale (SRS) is the patient's 
rating of the relation with the therapist and the degree 
of agreement on goals, methods, and treatment ap- 
proach, it has similarities to a treatment alliance rating. 
In the present study, the per protocol analysis showed 
that the PCOMS group had a higher score on SRS that 
was near to a p-value of 0.05. This could indicate that 
the feedback scales have an effect on an alliance-related 
dimension, which is encompassed by the SRS but not 
the TAS. The results on the SRS have to be interpreted 
with caution since the PCOMS group was exposed to 
the scales in the treatment sessions, and this might have 
influenced their answers. 

Since the most important factor affecting patient satis- 
faction is the relationship between patient and profes- 
sional [16], it is reasonable that the results on treatment 
alliance and patient satisfaction in the present study 
were consistent. The lack of effect on patient satisfaction 
is in line with previous investigations where clinicians 
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have been provided patient ratings of health status or 
quality of life [17,18]. 

Motivation 

The present study found that the PCOMS group had 
higher motivation for treatment than the treatment as 
usual group, six weeks after starting treatment. Patients' 
expectations of treatment effect have been shown by 
some to be important predictors of treatment outcome 
[55], while others have shown that motivation does not 
predict treatment result [56]. It has been argued that 
motivation is an internal patient factor which cannot be 
influenced, while others highlight social factors as im- 
portant for changing motivation [57]. Research has 
shown that motivation correlates with alliance at treat- 
ment start [58]. In the literature, motivation has been 
linked to setting and revising goals, encouraging the 
patient's view, and welcoming the patient's value system 
[57]. These factors are central when using patient feed- 
back scales. 

Treatment outcome 

Previous randomised controlled trials have shown that 
use of PCOMS was superior to treatment as usual after 
between 5 and 8 sessions [34-36]. These studies have 
used the ORS scale as the main outcome measure. How- 
ever, in the present study there were no short-term dif- 
ferences between the PCOMS group and the treatment 
as usual group regarding the ORS scores. This could be 
due to the shorter follow up time in this study. Never- 
theless, the mean ORS score for the PCOMS group fol- 
lowed the predicted score during the consultations, 
indicating that this group had good progress during 
treatment [30]. Furthermore, there was a clear change in 
the ORS scores from baseline to six week follow-up 
within the PCOMS group (mean difference = 3.53, 95% 
CI 0.99 to 6.06, p = 0.008), and not within the treatment 
as usual group (mean difference = 2.51, -0.41 to 5.43, p = 
0.090). A longer follow up time is needed to make more 
exact comparisons with other studies investigating the ef- 
fect of use of PCOMS. 

Conclusion 

The present study is the first randomised controlled trial 
investigating the short-term effect on treatment alliance 
and patient satisfaction of using the PCOMS scales in 
out-patient treatment. Compared to treatment as usual, 
there were no effects on alliance and satisfaction from 
using the scales six weeks after starting treatment. The 
PCOMS group had higher motivation for treatment. In 
addition, the per protocol analyses showed higher effect 
sizes. Future investigations in a larger study are therefore 
warranted. 
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